posted 08-13-2007 09:10 PM            

By now its no secret that we have been developing a new and improved version of the Objective Scoring System.

This has been a year-long project (read: obsession), and that began last August, following the APA conference in Las Vegas, and it has consumed hundreds of hours development and training time, and perhaps a few hundred hours of telephone conferencing.

Our initial design objectives were to redevelop the already robust OSS to handle more different types of examination. This required a substantial redesign of the mathematical transformations, and required re-training the normative data against which individual test data are compared for a decision.

We wanted our new OSS to be able to score any of the variety of recognized techniques that we all know and love, including tests with two to four relevant questions. We also wanted to be able to exploit the advantages of scoring three to five charts when possible, and we wanted a test that is not dependent upon an equivalent number of RQs and CQs.

To train the new algorithm, we used bootstrap resampling - a fascinating and computer intensive method of handling data (which would be darn-near impossible without fast modern computers). We made careful decisions about the use parametric and nonparametric statistical models, and the use of omnibus models where they commonly apply. We evaluated our decision alpha levels, including the use of Bonferonni corrections, and were able to incorporate advanced two-stage Senter rules into our decision models. We also have special rules intended to maximize sensitivity to deception while preserving balanced accuracy with screening exams.

During the last year we have validated and crossvalidated the new method on samples of ZCT cases, and a sample of cases with various techniques, drawn from the DACA (formerly DodPI) confirmed case archive. We have further crossvalidated the new algorithm on LEPET screening cases and two samples of PCSOT exams.

We intended from the outset for our new algorithm to be fully documented, using mathematical and statistical procedures that would be instantly recognizable to any university level or advanced statistician or researcher. We wanted our algorithm to adhere to and account for all of the necessary mathematical and data derived assumptions that factor into any responsible scoring algorithm, including combinatoric concerns related to inclusive rates with multi-facet and mixed issue examinations, and inflated alpha problems that result when completing multiple simultaneous significance tests.

One of our goals has been to place the algorithm in the public domain by publication, in the same manner as previous OSS versions. We have made all of our available docummentation available We also intended for our new algorithm to be free to anyone who wants to be able to use or study it. Some polygraph equipment developers have already begun to look at the new algorithm, and we have solicited feedback from a lot of examiners, and other people knowlegable about measurement testing and statistics.

Mostly, its getting harder and harder to be satisfied with "black-box" approaches to our scoring problems.

Also, we are not charging anything for it. A great deal of our development and validation experiments were completed using free and open-source software tool, So - to borrow from the open-source community - that's "free", as in "freedom" (do what you want with it), free as in "free speach" (read and say what you want about it), and free as in "free beer" (enjoy).

This has been a year-long project (read: obsession), and we've prepared three poster session presentations to illustrate the new algorithm at the upcoming APA conference.

We'll continue to make information available at:
http://www.oss3.info

Peace,

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 08-14-2007 11:01 AM            

Thanks for asking.

OSS-3 was trained on examination data acquired with Axciton system, and the Axciton system is capable of providing all of the physiological data to score OSS (all versions).

Our LEPET cases included both Lafayette and Axciton data. PCSOT data were obtained from Lafayette systems.

We used the Extract.exe tool, from John Harris (1994), to achieve all of the physiological measurements ala Kircher and Raskin (1988). Its a very small program, and runs quite fast.

Limestone provides a tool to make all of the Kircher measurements and export them to a .csv file for universal access to the data.

Lafayette has already developed the capacity to make Kircher measurements, because they have included an OSS-1/2 tool in their current software versions. I'm unsure whether they provide for the easy export of that data - which would be really cool for research and data analysis. Someone should ask them to do that, if they don't already.

Stoelting also has the capacity to make Kircher measurements, as the CPS-II system seems to use the same physiological features (duh) as OSS. Stoelting has also included an OSS-2 tool in their current software versions. We've offered our algorithm and any technical assistance to them at no cost (free beer). With Kircher's expertise we doubt they need much help from us. Stoelting doesn't seem to offer an easy way to export the Kircher measurements, but the data are included in their report output, which is stored on the computer as a .txt file. So, using text and string processing macros, we were able to automate the input of Kircher measurements from a sample of Stoelting cases to our OSS-3 laboratory model.

Axciton does not provide an OSS tool, and does not appear to provide access to the Kircher measurements. Instead they have limited their algorithm offerings to undocumented and proprietary methods. I think they should do both. OSS-2 appears to be about as robust as other algorithms in its intended usage. Like previous OSS versions the new version is also fully documented and non-proprietary, but is substantially more rubust in its field application, and appears to outperform the previous versions. The Kircher measurements hold a unique position in polygraph science, because they are the only physiological signals that are supported by published research and available documentation, and they hold relevance to both hand-scoring and automated scoring system.

Axciton has not adequately documented what physiological features their tool employs. I wondered about this some years ago, because their result output will report "probability" values over 100%. I certainly don't know everything about measurement and statistics, but 100+ appears to be certainty, not probability. Most probability models are not capable of reporting probability values of >=1. Logits (logistic regression) is an exception, as it can provide probability values in excess of 1. - but Axciton does not appear to use a logistic regression model, and Bruce wouldn't or couldn't explain in English what his software measured. They appear to use a cumulation of data, but there is no description of the features, scale or variance of the data - so it could just be smoke and mirrors and not real math.

Polyscore uses Logits, but wisely doesn't report probability values over 1. Polyscore also uses its own physiological signals, which were derived atheoretically and would be difficult for field examiners to appreciate. Polyscore does not appear to include the lower pneumo in its physiological signals or mathematical algorithm. (Please, someone tell me if I'm wrong about that.)

To score OSS-3 with the Axciton, one could print and obtain the Kircher measurements by hand, using plenometer and the procedures described by Dutton (2000).

If you have access to the Extract.exe tool, then Kircher measurements can be obtained quite quickly from Axciton data. I have some macros that automate the importing of Extract data to our OSS-3 laboratory model.

While OSS-3 could be calculated by hand, we don't imagine most people will lack the propeller-headedness necessary to attempt such foolishness. It really should be automated.

I understand Bruce's desire to protect intellectual property and sell stuff. There's nothing wrong with that. However, there is a more important concern about what it takes to develope a more mature form of polygraph science. That will require documentation, openness, and a balance of American values like freedom, free speach, and free beer (even though someone's paying for it), and accountability, alongside other worthwhile values like profiting from one's work.

OK, 'nuff for now. I'm off to do some paid work.

Peace,

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 08-15-2007 10:23 AM            

quote:

---

I'm getting a bit frustrated with Bruce and I'm about ready to demand that I be allowed to "look behind the curtain" or I'm going to just click my ruby slippers together and go back to Kansas (or is that Canada???)

---

I always loved that one-liner "pay no attention to that man behind the curtain."

Just breathe deep and you'll know the difference between Kansas and Canada. Everytime I drive across Kansas I can't habituate fast enough to smell of rotting grain - causing me to wonder, why do we have Kansas? Someone explained this to me once. It seems there is a time of year when they make something called "silage," which I believe fermented grain. That makes me wonder, how lonely or bored do you have to be to start thinking it might be fun to get some cows all hooched up? Makes that broke-back mountain thing sound kinda tame by comparison. (Actually, I've met some wonderful people from Kansas).

Thanks for looking at that stuff. Let us know if we can provide anything else.

Because we intend to hold no profit goals from IP, we have no problem showing every detail about how OSS-3 works. We also want examiners to be more conversant with the complexities that must be attended to in order to make a scoring algorithm work properly.

Even if most people don't care to examine the details of how it works. The fact that we could do so if we wanted to puts us in a more informed and empowered intellectual position.

After studying all this for the past year, I find that I have more confidence in the capabilities of a properly designed computer scoring tool. I have a better sense of the test results are capable of meaning, and what their limitations seem to be. I also find that I'm even less satisfied with any scoring tool that is undocumented and for which we don't know how (or even whether) the developers addressed certain complexities (like what happens to our decision alpha when we conduct multiple simultaneous significance tests, and how does the addition rule for dependent probability events affect our observed inconclusive rates with multi-facet and mixed issues exams).

It would certainly be possible to make a sorting algorithm based on whatever criteria we specify. It might even work with a fair degree of classification accuracy. Such an algorithm could even be evaluated empirically for its classification or decision accuracy. But, if the criteria are not supported by research data, and if the decision models are inconsistent with the principles of testing and inferential statistics, then its an arbitrary model, not a mathematical model, and would be vulnerable to criticism by people knowledgeable about generally accepted testing and measurement models. So, it might make sense not to show the details of that kind of tool.

Science is measurement, and measurement is math. Measurement of amorphous phenomena (for which there is no actual physical substance) is an event driven and inferential problem, and that means statistics. I think we should be able to show our math to any university researcher or statistician, and not get funny looks from them.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 08-15-2007 11:08 AM         

again, thanks

stat

posted 08-15-2007 11:45 AM            

You should know that this was a team effort. Mark Handler and I spent countless hours planning and problem solving over the telephone, and he has been embedded in every stage of development. All we did was solve some practical and mathematical problems and train the tool. Don Krapohl and Barry McManus created the original OSS tool, which is a good robust tool in its intended application. Just look through almost any polygraph bibliography and you will see that Don has been as diligent and persistent as anyone at asking sensible questions and seeking sensible answers to what makes polygraph work.

Don Krapohl, Gordon Barland, John Kircher, and Stuart Senter are really the people who articulated and gave us the principles and ideas that make the OSS-3 tool work.

Gordon Barland expressed the idea, in 1985, of using the cumulative normal distributions for DI and NDI subjects, as a means of providing probability estimates that are independent of base-rates and more easily generalizable, compared with other models, to field situations in which base-rates may vary widely. That idea seems to have gone largely un-noticed until about 1999, when Don Krapohl and Barry McManus finally put it to good use with OSS. Kraphol also gave us the R/C ratio, which is a really important dimmentionless transformation of differential reactivity, without which all kinds of measurement and standardization problems would be a serious pain in the arse. Of course John Kircher, and David Raskin (1988) did the discriminate analysis that gave us the Utah criteria and simple set of robust Kircher features on which both OSS (and apparently CPS-II) are built. Stuart Senter gave us a much improved understanding of decision rules, and some practical solutions, in the form of two-stage rules, to the combinatoric addition rule problems with spot scoring.

So, OSS-3 didn't come from no-where, and we are simply the construction crew that put existing good ideas together. Krapohl, Barland, Kircher, and Senter are the engineers.

Now to motorcycles. New bikes just don't have the appeal of older one's. But the ultra classic and electra glide models look really good. I'm spoiled for now, with the low cost and silly power of my old muscle bikes. I'd love to play with a Vmax or an early Venturer (same motor). The VMax is one of the longest production bikes. They look great, and hold up well. The only think is that, I'm spoiled by my big windshield, and all the Winnebago (Vetter) touring gear.

I was going to ride to APA, but I have a fouling #1 cylinder telling me my 28 year old muffler is about shot inside, though the chrome is still good. So, I'm trying to decide whether to sell my original mufflers on Ebay, and replace them with an aftermarket solution - or sawzall them off and fit a shiny set of brand new road king take-offs on the head-pipes. Sawing good chrome is a heartbreaking thing, but bad mufflers are an abomination. The harley pipes will require re-jetting the carbs because the big HD motors turn slower and move less air than a big old UJM.

I have enough adventure in the works for now, so I'll most likely drive. Now I have to get rid of those racing slicks (read: bald tires) on the car.

See you at APA.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)